fix(file-based): give each transferred file a unique staging path - #1078
Draft
devin-ai-integration[bot] wants to merge 1 commit into
Draft
fix(file-based): give each transferred file a unique staging path#1078devin-ai-integration[bot] wants to merge 1 commit into
devin-ai-integration[bot] wants to merge 1 commit into
Conversation
Two source files that resolve to the same relative path shared one staging file, so the second download overwrote the first and the destination failed with FileNotFoundException after consuming and deleting the first copy. Co-Authored-By: bot_apk <apk@cognition.ai>
Contributor
Author
🤖 Devin AI EngineerI'll be helping with this pull request! Here's what you should know: ✅ I will automatically:
Note: I can only respond to comments from users who have write access to this repository. ⚙️ Control Options:
|
👋 Greetings, Airbyte Team Member!Here are some helpful tips and reminders for your convenience. 💡 Show Tips and TricksTesting This CDK VersionYou can test this version of the CDK using the following: # Run the CLI from this branch:
uvx 'git+https://github.com/airbytehq/airbyte-python-cdk.git@devin/1785066655-unique-file-transfer-staging-paths#egg=airbyte-python-cdk[dev]' --help
# Update a connector to use the CDK from this branch ref:
cd airbyte-integrations/connectors/source-example
poe use-cdk-branch devin/1785066655-unique-file-transfer-staging-pathsPR Slash CommandsAirbyte Maintainers can execute the following slash commands on your PR:
|
1 task
Contributor
Author
|
CI status on the two red connector checks:
All CDK checks — Pytest (all Python versions), Ruff, MyPy, package build — are green. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
File transfer (
use_file_transfer) staged every source file at{staging_dir}/{source_file_relative_path}. When two source files resolve to the same relative path, they share one staging file, which breaks syncs:/staging/files/Untitled document.docx, emits a record referencing it.FileChunkTask→localFile.delete()).java.io.FileNotFoundException: /staging/files/Untitled document.docx (No such file or directory), or the destination reads bytes of B for record A.This is easy to hit on Google Drive (duplicate document names are allowed, and exported Google-native docs are named
{name}.{ext}) and on any source whenpreserve_directory_structureis disabled and the same file name exists in several folders.Fix: namespace the local staging path under a unique subdirectory, leaving
source_file_relative_path(and therefore the destination object key) unchanged:Because
LOCAL_FILE_PATHcomes fromAbstractFileBasedStreamReader._get_file_transfer_paths, this covers the baseupload()as well as the readers that call the helper themselves (source-google-drive,source-s3,source-microsoft-sharepoint) with no connector-side change.The
Check: source-google-driveCI job here fails ontest_upload_file, which asserts the exact staging path; the test-only update it needs is in airbytehq/airbyte#82754 (same forsource-microsoft-sharepoint;source-s3already asserts withendswith). That check should go green once the monorepo PR merges.Out of scope but worth flagging: duplicate relative paths still map to a single destination object key, so one file overwrites the other in the destination. That is inherent to path-mirroring naming and is not what makes the sync fail.
Resolves https://github.com/airbytehq/oncall/issues/12851:
Declarative-First Evaluation
Not applicable: the bug is in the Python CDK's file-based stream reader (staging path construction), not in a declarative connector manifest. No declarative feature can influence where the CDK stages a transferred file.
Test Coverage
test_staging_paths_are_unique_per_fileinunit_tests/sources/file_based/test_file_based_stream_reader.pyasserts that two files resolving to the same relative path get identicalfile_relative_pathbut distinctlocal_file_path(parametrized for the duplicate-name andpreserve_directory_structure: falsecases). It fails onmain(both calls return the same local path).test_preserve_sub_directories_scenariosupdated: it now asserts the local path is{staging_dir}/{unique_subdir}/{expected_relative_path}instead of a hardcoded path.poetry run pytest unit_tests/sources/file_based -q→ 796 passed, 1 skipped.ruff check,ruff format --check, andmypypass.Versioning
No connector version bump or changelog entry: this repo publishes from release notes, and
pyproject.tomlversion is computed dynamically. Non-breaking — no schema, spec, stream, or state change; only the ephemeral local staging path changes, and it is communicated to the destination viastaging_file_urlin the same record.Link to Devin session: https://app.devin.ai/sessions/8ef791c2602147a2951781e0af756828